Skip to content

feat: Optimize schema generation with in-memory parallel processing (53% speedup) - #1145

Closed
nathan-stender wants to merge 1 commit into
mainfrom
optimize-schema-generation-performance
Closed

feat: Optimize schema generation with in-memory parallel processing (53% speedup)#1145
nathan-stender wants to merge 1 commit into
mainfrom
optimize-schema-generation-performance

Conversation

@nathan-stender

Copy link
Copy Markdown
Collaborator

Summary

This PR optimizes the schema generation process that was identified as the major bottleneck in CI test runs.

Problem

  • The test_generate_schemas_runs_to_completion test was taking 183-207 seconds (3+ minutes)
  • This single test consumed ~35% of the entire test suite runtime
  • The bottleneck was sequential processing and heavy I/O operations (temp files, subprocess calls)

Solution

Implemented a new generate_schemas_optimized() function that:

  1. Processes schemas in parallel using ProcessPoolExecutor
  2. Generates and formats code in memory - no temp files
  3. Uses Python APIs directly (black, autoflake) instead of subprocess calls
  4. Batches all writes to the end, minimizing I/O operations

Performance Improvements

  • Test time reduced from ~180s to 84s (53% speedup)
  • CI test suite overall improvement: ~2-3 minutes saved per run
  • No functionality changes - just performance optimization

Technical Details

  • Added generate_schemas_optimized.py with the parallel implementation
  • Original generate_schemas() now defaults to using the optimized version
  • Can be disabled with use_optimized=False parameter for debugging
  • All existing tests pass without modification

Testing

✅ All tests passing
✅ Schema generation produces identical results
✅ Backwards compatible - original implementation still available

This addresses the most significant performance bottleneck in the test suite.

…53% speedup)

This addresses the major CI bottleneck where schema generation test was taking 3+ minutes.

Performance improvements:
- Test time reduced from ~180s to 84s (53% speedup)
- Parallel processing with ProcessPoolExecutor
- In-memory code generation and formatting
- Eliminates most temporary file operations

Technical changes:
* Added generate_schemas_optimized() for parallel schema processing
* Uses Python APIs for formatting code directly (black, autoflake)
* Generates models in memory before writing final results
* Defaults to using the optimized version unless disabled

Co-authored-by: Claude <assistant@anthropic.com>
@nathan-stender
nathan-stender requested a review from a team as a code owner March 14, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant